ddtrace: Rich performance monitoring in distributed systems

Through performance monitoring, application developers and maintainers can find software bugs, performance bottlenecks, and source of tail latency in their applications. In this paper, we develop a general-purpose performance monitoring library, ddtrace, designed to be easy to integrate with messageand Rpc-based systems which issue child requests as part of handling a single client request, such as the RAMCloud storage system. We demonstrate that ddtrace can analyze Rpc performance across multiple machines. The strengths of ddtrace are twofold. • Low overheads. 130 nanoseconds per interval of code measured, 5x faster than the next fastest trace logging library we could obtain and test. • Scriptability. Rich performance monitors can be written in a widely-used language (currently, C++) using the ddtrace API. Furthermore, these scripts are decoupled from the application, so the application is protected from faulty performance monitors and vice versa, and performance monitors can be reused across multiple applications built with ddtrace support.